Information processing apparatus, information processing system, and information processing method

ABSTRACT

In an information processing apparatus  200 , a communication portion  250  acquires a data stream of a photographed image in a wireless communication manner. A packet analyzing portion  252  detects a lost packet, and a retransmission request producing portion  254  issues a retransmission request for a packet within a predetermined allowable time. Along with expiration of the allowable time, a lacked information producing portion  258  produces lacked information in which data lacked due to the packet loss is associated with a position on an image plane. A decoding portion  260  decodes the data associated with the photographed image and stored in the buffer  256 , and substitutes invalid data for the lacked portion. An image analyzing portion  262  differentiates the lacked portion from other portions, and performs analysis processing. An information processing portion  264  and an output data producing portion  266  execute information processing by utilizing the analysis result, and produce output data, respectively.

TECHNICAL FIELD

The present invention relates to an information processing apparatus andan information processing system each of which executes informationprocessing by utilizing a photographed image, and an informationprocessing method carried out in those.

BACKGROUND ART

There is known a game in which a user is photographed with a videocamera, and the resulting image is replaced with another image to bedisplayed on a display (for example, refer to PTL 1). In addition, thereis also known a user interface system in which a motion of a mouth or ahand photographed with a video camera is received as a manipulationinstruction of an application. In such a way, the technique with which areal world is photographed and a virtual world responding to a motion ofthe real world is displayed, or some sort of information processing isexecuted is utilized in a wide range of fields from a small portableterminal up to a leisure facility irrespective of the scale thereof.

A system with which a panorama image is displayed on a head mounteddisplay, and when a user wearing the head mounted display rotateshis/her head, the panorama image responding to a gaze direction isdisplayed is also developed as an example thereof. The head mounteddisplay is utilized, resulting in that a sense of immersion in the imagecan be increased, or the operationability of an application such as agame can be enhanced. In addition, a walk through system with which theuser wearing the head mounted display physically moves, resulting inthat the user can virtually walk around within the space displayed asthe image is also developed.

CITATION LIST Patent Literature

[PTL 1]

EP 0999518 A1

SUMMARY Technical Problems

For realizing the picture expression having the presence, or executingthe information processing with high accuracy, it is desirable toincrease the resolution or the frame rate of the photographed image. Onthe other hand, if the number of data sizes which should be handled insuch a manner is increased, then, a load applied to data transmission orimage analysis is increased, and a period of time required until theimage display is increased. As a result, there is caused the shiftbetween the actual motion and the display, and so forth, and thus theuser may be given a sense of discomfort in some cases. In addition, thetransmission band is oppressed, resulting in that a lack of the databecomes easy to occur, and the accuracy of the image analysis may becomerather worse in some cases.

The present invention has been made in the light of such problems, andit is therefore an object of the present invention to provide atechnique with which a satisfactory processing result can be acquiredwithout injuring currency in information processing followed bytransmission and an analysis of data associated with a photographedimage.

Solution to Problems

In order to solve the problems described above, a certain aspect of thepresent invention relates to an information processing apparatus. Theinformation processing apparatus is characterized by including acommunication portion, a lacked information producing portion, an imageanalyzing portion, and an output data producing portion. In this case,the communication portion serves to establish a communication with animage pickup device and acquire data associated with a photographedimage. The lacked information producing portion serves to produce lackedinformation with which data lacked as a result of the communication, anda position on the photographed image are associated with each other. Theimage analyzing portion serves to differentiate a portion in which thedata is lacked from other portions by referring to the lackedinformation, and analyze the photographed image. The data producingportion serves to produce output data based on an analysis result andoutput the output data.

Another aspect of the present invention relates to an informationprocessing system. The information processing system is characterized byincluding a head mounted display provided with an image pickup device,and an information processing apparatus configured to establish acommunication with the head mounted display, produce a display image andcause the head mounted display to display the display image. Theinformation processing apparatus is provided with a communicationportion, a lacked information producing portion, an image analyzingportion, and an output data producing portion. In this case, thecommunication portion serves to acquire data associated with aphotographed image from the head mounted display. The lacked informationproducing portion serves to produce lacked information with which datalacked as a result of the communication, and a position on thephotographed image are associated with each other. The image analyzingportion serves to differentiate a portion in which data is lacked fromother portions by referring to the lacked information, and analyze thephotographed image. The output data producing portion serves to produceoutput data based on an analysis result and output the output data.

Still another aspect of the present invention relates to an informationprocessing method. The information processing method is characterized byincluding, by an information processing apparatus, a step ofestablishing a communication with an image pickup device and acquiringdata associated with a photographed image, a step of producing lackedinformation in which data lacked as a result of the communication isassociated with a position on the photographed image, a step ofdifferentiating a portion in which the data is lacked from otherportions by referring to the lacked information, thereby carrying out ananalysis of the photographed image, and a step of producing output databased on a result of the analysis and outputting the output data to adisplay device.

It should be noted that an arbitrary combination of constituent elementsdescribed above, and results obtained by converting an expression of thepresent invention among a method, an apparatus, a system, a computerprogram, a data structure, a recording medium, and the like are valid interms of aspects of the present invention.

Advantageous Effect of Invention

According to the present invention, in the information processingfollowed by the transmission and the analysis of the data associatedwith the photographed image, the satisfactory processing results can beacquired without injuring the currency.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an external view of a head mounted display of an embodiment.

FIG. 2 is a block diagram depicting a circuit configuration of the headmounted display of the embodiment.

FIG. 3 is a view depicting a configuration of an information processingsystem of the embodiment.

FIG. 4 is a block diagram depicting an internal circuit configuration ofan information processing apparatus of the embodiment.

FIG. 5 is a view schematically depicting a data structure when the headmounted display in the embodiment transmits an image photographed with acamera.

FIG. 6 is a view schematically depicting a data structure of an imagewhich the information processing apparatus in the embodiment receivesfrom the head mounted display.

FIG. 7 is a graphical representation quantitatively depicting a fallingrate of data with respect to a time allowable for retransmissionprocessing in image data transmission of the embodiment.

FIG. 8 is a block diagram depicting a configuration of functional blocksof the head mounted display in the embodiment.

FIG. 9 is a block diagram depicting a configuration of functional blocksof the information processing apparatus in the embodiment.

FIG. 10 is a diagram depicting an example of a data structure of apacket which is transmitted from the head mounted display to theinformation processing apparatus in the embodiment.

FIG. 11 is a diagram exemplifying a data structure of lacked informationwhich a lacked information producing portion of the informationprocessing apparatus produces in the embodiment.

FIG. 12 is a diagram exemplifying a data structure of lacked informationwhich the lacked information producing portion of the informationprocessing apparatus produces in the embodiment.

FIG. 13 is a diagram exemplifying a data structure of lacked informationwhich the lacked information producing portion of the informationprocessing apparatus produces in the embodiment.

FIG. 14 is a graphical representation explaining a relationship betweenan allowable time and analysis accuracy in the image data transmissionin the embodiment.

FIG. 15 is a block diagram depicting a configuration of functionalblocks of the information processing apparatus having functions ofadjusting a retransmission allowable time of a lost packet due toaccuracy of image analysis in the embodiment.

FIG. 16 is a block diagram depicting a configuration of the functionalblocks of the head mounted display having functions of adjusting aninternal processing condition by the retransmission allowable time ofthe lost packet in the embodiment.

FIG. 17 is a flow chart depicting a processing procedure for producingoutput data based on data associated with the photographed image by theinformation processing apparatus in the embodiment.

DESCRIPTION OF EMBODIMENT

In this embodiment, in a system in which information processing isexecuted by using an image photographed with an image pickup device, anda result of the information processing is immediately displayed, even ifdata is lacked at the time of transmission, the maximum performance canbe exhibited. In this regard, although the form of the image pickupdevice or a display device is not especially limited, in this case, adescription will be given by exemplifying a head mounted displayprovided with a camera. Drawing and displaying an image in which a fieldof view is changed so as to respond to a motion of a head of a userwearing the head mounted display by utilizing an image photographed bythe head mounted display are a suitable example from a viewpoint of theseverity of a time constraint. However, even if the image pickup deviceand the display device have different chassis, the embodiment can besimilarly applied to this case.

FIG. 1 depicts an example of an external shape of a head mounted displayin an embodiment. In this example, a head mounted display 100 includesan output mechanism portion 102 and a mounting mechanism portion 104.The mounting mechanism portion 104 includes a mounting band 106 whichmakes a circuit of a head of a user with the user wearing the mountingband 106 to realize the fixing of the apparatus. The mounting band 106is made of a raw material or has a structure which can be adjusted inlength in response to the circumference of the head of the user. Forexample, the mounting band 106 may be composed of an elastic body suchas a rubber, or may utilize a buckle, a gear or the like.

The output mechanism portion 102 includes a chassis 108 having such ashape as to cover the both eyes in a state in which the user wears thehead mounted display 100, and is provided with a display panel in aninside thereof in such a way that the display panel directly faces theeyes at the time of the wearing. The display panel is realized by aliquid crystal panel, an organic Electroluminescence (EL) panel, or thelike. The chassis 108 is further provided with a pair of lenses in theinside thereof which are located between the display panel and the eyesof the user at the time of the wearing of the head mounted display 100,and by which an image can be visually recognized in a wide field ofview. In addition, the head mounted display 100 may be further providedwith speakers or earphones in positions corresponding to the ears of theuser at the time of the wearing.

The head mounted display 100 is provided with a camera 110 in a frontsurface of the output mechanism portion 102. The camera 110 is providedwith an image pickup element such as a Charge Coupled Device (CCD) or aComplementary Metal Oxide Semiconductor (CMOS), and photographs the realspace at a predetermined frame rate in a field of view corresponding toa direction of a face of the user who wears the head mounted display100. It should be noted that only one camera 110 may be provided in thefront surface of the head mounted display 100, or two cameras may beprovided as stereoscopic cameras which are arranged in the right andleft sides so as to have a known interval. In addition, the arrangementof the cameras 110 is not especially limited.

The image photographed by the camera 110 can be used as at least a partof the display image in the head mounted display 100. In additionthereto, the image concerned can also be used for input data for theimage analysis necessary for the production of the virtual world. Forexample, if the photographed image is made the display image as it is,then, the user undergoes the same state as that in which the userdirectly sees the real space right in front of his/her eyes. Inaddition, an object which stays on the real object such as a disk lyingwithin the field of view or interactions with the real object is drawnon the photographed image to be made the display image, thereby enablingthe Augmented Reality (AR) to be realized.

Moreover, the position and posture of the user who wears the headmounted display 100 are specified from the photographed image, and thefield of view is changed so as to correspond to the position and postureof the user to draw the virtual world, thereby also enabling the VirtualReality (VR) to be realized. The general technique such as VisualSimultaneous Localization And Mapping (v-SLAM) can be applied to thetechnique for estimating the position and posture of the camera from thephotographed image. An angle of rotation and an inclination of the headmay be measured by a motion sensor built in the head mounted display 100or an external motion sensor. The analysis result of the photographedimage, and the measured value by the motion sensor may be utilized in amutually complementary manner.

FIG. 2 depicts a circuit configuration of the head mounted display 100.A control portion 10 is a main processor for processing signals such asan image signal and a sensor signal, or an instruction or data, therebyoutputting the resulting signals, instruction or data. The camera 110supplies the data associated with the photographed image to the controlportion 10. A display 30 is composed of a liquid crystal display or thelike, receives the image signal from the control portion 10, anddisplays an image associated with the image signal.

A communication control portion 40 transmits the data inputted theretofrom the control portion 10 to the outside through a wired or wirelesscommunication via a network adapter 42 or an antenna 44. Thecommunication control portion 40 also receives the data from the outsidethrough the wired or wireless communication via the network adapter 42or the antenna 44, and outputs the data thus received to the controlportion 10. A storage portion 50 temporarily stores the data, aparameter, a manipulation signal, and the like which the control portion10 processes.

A motion sensor 64 detects posture information associated with an angleof rotation, an inclination and the like of the head mounted display100. The motion sensor 64 is realized by suitably combining a gyrosensor, an acceleration sensor, a geomagnetic sensor, and the like withone another. An external input/output terminal interface 70 is aninterface for connection of peripheral units such as a Universal SerialBus (USB) controller. An external memory 72 is an external memory suchas a flash memory. The control portion 10 can supply the image or sounddata to the display 30 or a headphone (not depicted), thereby outputtingthe image or the sound, or can supply the image or the sound data to thecommunication control portion 40, thereby transmitting the image or thesound data thus supplied to the outside.

FIG. 3 is a view depicting a configuration of an information processingsystem according to the embodiment. The head mounted display 100establishes a communication with an information processing apparatus 200in accordance with a communication protocol such as a Bluetooth(registered trademark) protocol or an institute of Electrical andElectronic Engineers (IEEE) 802.11 protocol via the antenna 44 of FIG.2. The information processing apparatus 200 may be further connected toa server through a network. In this case, the server may present anonline application such as a game in which a plurality of users canparticipate through the network to the information processing apparatus200. The head mounted display 100 may be connected to a computer or aportable terminal instead of being connected to the informationprocessing apparatus 200.

The information processing apparatus 200 basically repetitively executesthe processing in which after the data associated with the imagephotographed with the camera 110 of the head mounted display 100 issuccessively received to be subjected to predetermined processing, theimage to be displayed is produced to be transmitted to the head mounteddisplay 100 every frame. As a result, the various images such as AR andVR are displayed in the field of view responding to the direction of theface of the user on the head mounted display 100. It should be notedthat the game, the virtual expression, the appreciation of the movingimage, and the like are variously considered as the ultimate aim of suchdisplay.

FIG. 4 depicts a configuration of an internal circuit of the informationprocessing apparatus 200. The information processing apparatus 200includes a Central Processing Unit (CPU) 222, a Graphics Processing Unit(GPU) 224, and a main memory 226. These portions are connected to oneanother through a bus 230. An input/output interface 228 is furtherconnected to the bus 230.

To the input/output interface 228, a communication portion 232, astorage portion 234 such as a hard disc drive or a non-volatile memory,an outputting portion 236, an input portion 238, and a recording mediumdrive portion 240 are connected. In this case, the communication portion232 is composed of a peripheral unit interface such as the USE or theIEEE1394, or a network interface such as a wired or wireless Local AreaNetwork (LAN). The outputting portion 236 serves to output the dataassociated with the image or sound to a display device or a speaker (notdepicted). The input Portion 238 serves to receive as an input thereofthe data from an input device (not depicted). The recording medium driveportion 240 serves to drive a removable recording medium such as amagnetic disk, an optical disc or a semiconductor memory.

The CPU 222 executes an operating system stored in the storage portion234, thereby controlling the whole of the information processingapparatus 200. The CPU 222 also executes the various kinds of programswhich are read out from the removable recording medium to be loaded intothe main memory 226, or are downloaded through the communication portion232. The GPU 224 has a function of a geometry engine, and a function ofa rendering processor, and executes drawing processing in accordancewith a drawing instruction issued from the CPU 222. The main memory 226is composed of a Random Access Memory (RAM), and stores therein theprogram and data required for the processing.

FIG. 5 schematically depicts a data structure when the head mounteddisplay 100 transmits the image photographed with the camera 110. In thecase where the camera 110 is composed of a stereoscopic camera, a pairof photographed image 80 a in the left point of view and photographedimage 80 b in the right point of view is acquired at a predeterminedframe rate. The head mounted display 100 packetizes the acquired imagedata in order from the upper left of the image in a predeterminedtransmission unit so as to correspond to the order of scanning in theimage pickup element, and transmits the image data thus packetized tothe information processing apparatus 200.

The figure represents that the image data is packetized in thetransmission unit of L1, L2, L3, . . . from the upper left of thephotographed image 80 a in the left point of view, and R1, R2, R3, . . .from the upper left of the photographed image 80 b in the right point ofview.

The data in the transmission units is subjected to the compressioncoding in a predetermined format, and the data associated with thephotographed image 80 a in the left point of view, and the photographedimage 80 b in the right point of view is alternately sent. That is, thedata in the transmission unit of L1, R1, L2, R2, L3, R3, . . . istransmitted as a data stream in this order.

It should be noted that the number of rows and the number of columns inthe pixels constituting the transmission units are determined based onthe affinity with the coding system adopted. In addition, in theillustrated example, for simplification, the area of the image in thetransmission unit, and the data size of the image coded and contained inone packet are both equally expressed. However, in the case where thevariable length coding system is adopted, one of them may be inequal.

FIG. 6 schematically depicts a data structure of the image which theinformation processing apparatus 200 receives from the head mounteddisplay 100. As described in FIG. 5, the data associated with thephotographed image in the left point of view, and the data associatedwith the photographed image in the right point of view are alternatelytransmitted in the predetermined transmission unit from the head mounteddisplay 100. Therefore, properly speaking, as depicted in the upperstage of the figure, the data in the transmission unit of L1, R1, L2,R2, L3, R3, . . . acquired as the data stream in this order.

The information processing apparatus 200 decodes the data which issequentially transmitted in such a manner, and develops the decoded datain the buffer memory so as to be associated with the original positionson the image plane, thereby restoring the original photographed image,that is, the photographed image 82 a in the left point of view, and thephotographed image 82 b in the right point of view. However, the packetis lost during the transmission due to the communication situation orthe like. The figure depicts the fact that the transmission units R2 andL3 are not transmitted in the form of the hatching. At this time, thephotographed images 82 a and 82 b restored are naturally imperfect.

As measures taken for such a case, heretofore, there has been a functionof an Automatic repeat-request (ARQ) for requesting the transmissionsource to retransmit the lost packet. The ARQ is mounted as one of theerror correction methods to the communication protocol such as aReal-time Transport Protocol (RTP) or a Transmission Control Protocol(TCP). If the retransmitted packet can be acquired by this function,then, the error correction including the loss can be reliably carriedout. On the other hand, a time shall be consumed due to the transmissionof the retransmission request signal and the packet retransmission ofthe data concerned. In addition, it is also possible that even in such acase, the retransmitted packet is lost again.

FIG. 7 qualitatively depicts a falling rate of the data with respect toa time allowable for the retransmission processing when the image datais transmitted. As depicted in the figure, as the allowable time islonger, the margin occurs in the individual retransmission processing,and also the probability that it is possible to cope with a plurality oftimes of the packet losses in the same data is increased. As a result,the falling rate of the data is reduced. On the other hand, in the casewhere it is supposed that after like the embodiment, some sort ofinformation processing is executed by using the photographed image, theimage which moves so as to correspond to the actual motion is displayed,a time allowable for the data transmission is very short.

Then, even if the allowable time is limited, the image data which istransmitted for that time is utilized to the fullest, and thus the imageanalysis is properly carried out. Specifically, the portion in which thedata is lacked on the image is signalized, resulting in that in theimage analysis or the subsequent information processing, the portionconcerned can be handled so as to be distinguished from other portions.In general, the processing such as ARQ pertaining to the datatransmission is controlled independently of other pieces of processingby the mutual communication mechanisms. In the embodiment, the parameterpertaining to such transmission is organically coupled to the analysisfunction which is mounted to the application such as the game, therebycausing the balance between the currency including the datatransmission, and the accuracy of the processing to be satisfactory. Inaddition, by utilizing such a mechanism, the allowable time can beadjusted from a viewpoint of the processing accuracy, and the algorithmof the processing can be suitably switched in response to the fallingrate of the data decided by the allowable time.

FIG. 8 depicts a configuration of functional blocks of the head mounteddisplay 100 in the embodiment. The functional block depicted in FIG. 8and FIG. 9 which will be described later can be realized by the variouskinds of mechanisms depicted in FIG. 2 or FIG. 4 in terms of thehardware, and can be realized by a program which is loaded from therecording medium or the like into the memory, and which exhibits thefunctions such as the data analysis function and the image processingfunction in terms of the software. Therefore, it is understood by aperson skilled in the art that these functional blocks can be realizedin the various forms by only the hardware, only the software, or thecombination thereof, and is by no means limited to any of those.

The head mounted display 100 includes a data acquiring portion 120, acoding portion 122, a buffer 124, a communication portion 126, aretransmission control portion 128, and a data outputting portion 130.In this case, the data acquiring portion 120 serves to acquire thephotographed image from the camera 110. The coding portion 122 serves tocode the photographed image. The buffer 124 serves to temporarily storethe coded data. The communication portion 126 is an interface throughwhich the data is transmitted/received to/from the informationprocessing apparatus 200. The retransmission control portion 128 servesto process the retransmission request from the information processingapparatus 200. The data outputting portion 130 serves to output thedisplay image or the like.

The data acquiring portion 120 is composed of the camera 110, thecontrol portion 10, and the like of FIG. 2, and acquires the dataassociated with the images which are obtained by photographing the spacein the field of view at the predetermined frame rate. In the case wherethe camera 110 is composed of the stereoscopic cameras as depicted inFIG. 1, the photographed images become a pair of parallax images havingthe field of view corresponding to the left eye and the right eye of theuser who wears the head mounted display 100. However, the embodiment isby no means limited thereto in gist, and the moving image at one pointof view may be photographed with a monocular camera. The data acquiringportion 120 supplies the pixel values in order of the pixels from whichthe pixel values can be acquired to the coding portion 122.

The coding portion 122 is composed of the control portion 10 and thelike, and subjects the data associated with the photographed imageacquired by the data acquiring portion 120 to the compression coding inaccordance with the predetermined coding system such as JointPhotographic Experts Group (JPEG). The buffer 124 is composed of thestorage portion 50, and temporarily preserves the data associated withthe photographed image subjected to the compression coding so as toprepare for the retransmission request. The communication portion 126 iscomposed of the communication control portion 40, the antenna 44, andthe like, and packetizes the data associated with the photographed imagesubjected to the compression coding every transmission unit, andtransmits the resulting data to the information processing apparatus200. As depicted in FIG. 5, the data is coded in order of the pieces ofdata which can be acquired by the image pickup element, packetizes theresulting data in the transmission unit to transmit the resulting data,thereby enabling the photographed image to be transmitted at a lowlatency.

The communication portion 126 also receives the retransmission requestfor the packet concerned which is issued when the information processingapparatus 200 detects the packet loss, and informs the retransmissioncontrol portion 128 of this effect. The retransmission control portion128 reads out the data in the transmission unit corresponding to thepacket specified in the retransmission request from the buffer 124, andsupplies the resulting data to the transmission portion 126. Thetransmission portion 126 packetizes the data concerned again to transmitthe resulting data, thereby realizing the retransmission processing forthe lost packet. The transmission portion 126 further receives the dataassociated with the display image or the output sound which istransmitted from the information processing apparatus 200, and suppliesthe data thus received to the data outputting portion 130. The dataoutputting portion 130 is composed of the control portion 10, thedisplay 30, and the like, and carries out the display of the displayimage, or the sound output.

FIG. 9 depicts a configuration of functional blocks of the informationprocessing apparatus 200 in the embodiment. The information processingapparatus 200 includes a communication portion 250, a packet analyzingportion 252, a retransmission request producing portion 254, a buffer256, a lacked information producing portion 258, a decoding portion 260,an image analyzing portion 262, an information processing portion 264,and an output data producing portion 266. In this case, thecommunication portion 250 is an interface through which the data istransmitted/received to/from the head mounted display 100. The packetanalyzing portion 252 serves to analyze the packet transmitted from thehead mounted display 100. The retransmission request producing portion254 serves to produce the retransmission request for the lost packet.The buffer 256 serves to temporarily store the data associated with thephotographed image transmitted thereto. The lacked information producingportion 258 serves to produce information pertaining to the occurrencesituation of the data lack. The decoding portion 260 serves to decodethe data associated with the photographed image. The image analyzingportion 262 serves to carry out the predetermined analysis for thedecoded photographed image. The information processing portion 264serves to execute the predetermined information processing by using theanalysis result. In addition, the output data producing portion 266serves to produce the data to be outputted such as the display image.

The communication portion 250 is composed of the communication portion232, the CPU 222, and the like of FIG. 4, and receives the packet whichis transmitted from the head mounted display 100 and which includes thedata associated with the photographed image. In addition, thecommunication portion 250 transmits the retransmission request for thepacket which is produced by the retransmission request producing portion254, and the data associated with the display image or the output soundwhich is produced by the output data producing portion 266 to the headmounted display 100.

The packet analyzing portion 252 is composed of the CPU 222 and thelike, and successively acquires the packets transmitted thereto from thehead mounted display 100 and analyzes the packets thus acquired.Specifically, the packet analyzing portion 252 detects the packet whichis lost in the data transmission based on the identification information(hereinafter, referred to as “the packet Identification (ID)”) givenevery packet. In addition, the packet analyzing portion 252 separatesthe header information containing the packet ID and the like, and thedata main body from each other, and stores the latter in the buffer 256so as to be associated with the position on the image plane.

In the case where the loss of the packet occurs, the packet analyzingportion 252 informs the retransmission request producing portion 254 ofthe packet ID concerned. The retransmission request producing portion254 is composed of the CPU 222 and the like, and produces theretransmission request containing the lost packet ID. The retransmissionrequest concerned is transmitted to the head mounted display 100 throughthe communication portion 250. In the example of FIG. 6, the packetcorresponding to the transmission unit such as “R2” or “L3” is specifiedto issue the retransmission request.

Although the retransmission request producing portion 254 may repeat theretransmission request until the lost packet can be acquired, theretransmission request producing portion 254 determines the time out ifa predetermined allowable time elapses with the specified packet beingnot acquired, and thereafter, the retransmission request producingportion 254 does not make out the retransmission request about thepacket concerned. For this reason, the retransmission request producingportion 254 holds a set value of the allowable time in an internalmemory. The lacked information producing portion 258 is composed of theCPU 222 and the like, and produces the information pertaining to thedata which could not be acquired even when the allowable time describedabove elapses (hereinafter, referred to as “the lacked information”).

The lacked information basically represents which position on the imageto what degree the data lack occurs. For this extent, the lackedinformation producing portion 258 successively acquires the informationpertaining to the data in the lacked transmission unit from the packetanalyzing portion 252, and acquires a timing at which the allowable timeexpires from the retransmission request producing portion 254, therebyfinally specifying the portion in which the data was lacked. Since inthe more forward portion in transmission order, the allowable time earlyexpires, the lacked information can be produced synchronously with theprocessing start of the corresponding area. The lacked information thusproduced is successively supplied to the decoding portion 260.

The decoding portion 260 is composed of the CPU 222 and the like, andsuccessively reads out the data main body from the buffer 256 to decodethe data main body, thereby restoring the photographed image. Thetransition of the data as depicted in FIG. 6 is successively realized bythe packet analyzing portion 252, the buffer 256, and the decodingportion 260. In this case, in the lacked information acquired from thelacked information producing portion 258, predetermined invalid data issubstituted for the pixel which is proved as the lacked portion. Thedecoding portion 260 associates the data associated with the restoredphotographed image with the lacked information, and supplies theresulting data to the image analyzing portion 262.

The image analyzing portion 262 is composed of the CPU 222 and the like,and carries out the predetermined image analysis for the data associatedwith the photographed image restored by the decoding portion 260. Thecontents of the analysis carried out here are variously considereddepending on the contents of the information processing in thesubsequent stage, the object of the photographing, the contents of thedisplay, and the like. For example, for the purpose of acquiring theposition and posture of the head of the user, the v-SLAM described aboveis carried out, and for the purpose of acquiring the position in thereal space of the subject, the stereoscopic matching is carried out byusing the right and left parallax images to produce a depth image. Thedepth image is an image in which a distance of a subject from a camerais expressed in the form of a pixel value of a corresponding image on aphotographed image. The depth image is obtained by extractingcorresponding points from the parallax image, and calculating a distancein accordance with the principle of triangulation based on the parallaxbetween both of them.

Alternatively, the three-dimensional modeling of the real object lyingin the photographed space is carried out by using the depth image. Thereal object is modeled as the object in the computationalthree-dimensional space, thereby enabling AR in which the virtual objectand the real object interact with each other to be realized.Alternatively, an image of the real object having a predetermined shapeor color may be detected based on the template matching, or people orgoods may be recognized based on the feature of the details such as theface.

The image analyzing portion 262 suitably differentiates in processingcontents the portion in which the data is lacked due to the packet lossfrom other areas by referring to the lacked information in such an imageanalysis. For example, the portion in which the data is lacked isexcluded from the area of the analysis object such as the object of thecorresponding point retrieval for the production of the depth image, orthe subject of the template matching for the detection of the subject.Alternatively, after the analysis is carried out similarly to othercases, the result about the portion in which the data is lacked may beprevented from being outputted, and so forth, or additional informationhaving a possibility of the low accuracy in the portion concerned may below may be produced.

It should be noted here that “the portion in which the data is lacked”may mean a portion in which there is truly no data, or may mean an areain which a falling rate when the image plane is divided into parts inaccordance with a predetermined rule is higher than a predeterminedthreshold value. In any case, the image analyzing portion 262 recognizesthe portion in which the data is lacked and can differentiate theanalysis processing itself or a utilization form of the analysis resultfrom other cases, resulting in that a change for the worse of theprocessing accuracy due to the lack of the data can be kept to aminimum.

That is, the area which is analyzed with the low accuracy because nodata is obtained is handled similarly to the analysis results of otherareas, resulting in that the accuracy of the processing can be preventedfrom being reduced in the entire image. In addition, the portion inwhich the data is lacked is excluded from the analysis object such asthe corresponding point retrieval, resulting in that the unnecessaryprocessing in which the reflection on the result is not preferable canbe reduced. These ways of handling are guaranteed, resulting in thateven when the allowable time for the retransmission request isshortened, the bad influence is suppressed, and thus the accuracy andthe currency are compatible with each other.

The information processing portion 264 is composed of the CPU 222 andthe like, and executes the predetermined information processing such asthe game by utilizing the result of the image analysis. As describedabove, the contents of the predetermined information processing is notespecially limited. The information processing Portion 264 maydifferentiate the analysis result of the portion exhibited by the imageanalyzing portion 262 from the reason that the portion has a possibilityof the low analysis accuracy due to the data lack from other cases. Forexample, the subject in which an edge is contained in such a portion isexcluded from the calculation object, or an object is not drawn in sucha portion. As a result, the change for the worse of the analysisaccuracy of a Partial area can be prevented from being propagated toother processing, or the gaze of the user can be induced to anotherportion.

The output data producing portion 266 is composed of the CPU 222, theGPU 224 and the like, and produces the data associated with the image tobe displayed or the sound to be outputted on or to the head mounteddisplay 100 as a result of the processing executed by the informationprocessing portion 264. For example, the output data producing portion266 draws the virtual world in which the field of view is changed inresponse to the motion of the head of the user, or produces the sounddata in the virtual world concerned. The data thus produced issuccessively transmitted to the head mounted display 100 through thecommunication portion 250.

FIG. 10 depicts an example of a data structure of a product which istransmitted from the head mounted display 100 to the informationprocessing apparatus 200 in the embodiment. A packet 90 depicted in thefigure represents one of the packet columns depicted in the upper stageof FIG. 5 and FIG. 6. Thus, the data for one transmission unit of thephotographed image is transmitted based on the packet 90. The packet 90is composed of a packet information portion 92 and an image data portion94. The packet information portion 92 contains so-called headerinformation such as “packet ID,” “packet length,” “transmission time,”“Packet ID” is identification information which is uniquely given to allthe packets produced for one frame of the photographed image. Forexample, the packet ID may be numbers of the ascending order such as“0,” “1,” “2,” . . . which are given in the raster order to thetransmission unit, or may be two-dimensional position coordinates in theimage plane. Alternatively, the packet ID may be the head addresses ofthe transmission units when the image is developed in the memory. Inaddition, as depicted in FIG. 5 and FIG. 6, in the case where thephotographed image of the left point of view, and the photographed imageof the right point of view are alternately sent, moreover, an identifierfor distinguishing which of the images the packet belongs to is added tothe packet ID.

“Packet length” is a data size of the packet 90. “Transmission time” isa time stamp exhibiting the time at which the packet 90 is transmittedfrom the head mounted display 100. In addition thereto, other pieces ofinformation may be suitably contained in the packet information portion92 depending on the adapted communication protocol.

The data, in the transmission unit of the photographed image, which issubjected to the compression coding by the coding portion 122 of thehead mounted display 100 is stored in the image data portion 94. Thepacket analyzing Portion 252 divides the packet 90 transmitted theretointo the packet information portion 92 and the image data portion 94,and stores the data of the latter in the buffer 256 so as to beassociated with the position on the image. The information pertaining tothe grant rules of the packet ID is shared with the head mounted display100, thereby enabling the positions on the image of the pieces of datato be specified based on the packet ID. Therefore, the data within thepacket which can be acquired without being lost is stored in the buffer256 so as to be associated with the position on the image.

The packet analyzing portion 252 also confirms the packet ID of thepacket information portion 92 in comparison with the transmitted order,thereby detecting the loss of the packet. For example, in the case wherethe numbers of the ascending order are given as the packet ID, themissing can be detected by comparing the arrival order of the packets,and the packet ID with each other. The packet analyzing portion 252informs the retransmission request producing portion 254 of the ID ofthe lost packet detected in such a manner, thereby producing theretransmission request. The lacked information producing portion 258 isalso informed of the information associated with the lost packet or thenewest information Pertaining to the position on the image of the dataassociated with that information, thereby producing the lackedinformation with the expiration of the allowable time.

FIG. 11 exemplifies a data structure of the lacked information which isproduced by the lacked information producing portion 258 of theinformation processing apparatus 200. This example directly exhibits anaddress of the area, in which the data corresponding to the lost packetshould be stored, of the storage area 302 of the image data contained inthe buffer 256. That is, a lacked information 300 is composed of “headaddress” and “data length” of the lacked portion. For the lackedportion, the data transmitted in one packet, that is, the transmissionunit is recognized as a minimum particle size. Then, if the packets arecontinuously lost, then, a numerical value of “data length” in thelacked information 300 is increased.

This example exhibits that the data of (3001, 1500), (13501, 3000),(21001, 1500), . . . in order of (head address, data length) is lacked.If this is expressed in the storage area 302 of the image data, then, itis understood that the hatched portions, that is, the data in theportions of 1500 bytes of 3001 bytes to 4500 bytes, 3000 bytes of 13501bytes to 16500 bytes, 1500 bytes of 21001 bytes to 26500 bytes, . . . islacked. Since the storage area 302 of the image data is associated withthe image plane, the exhibition of the address of the storage area shallthus specify the corresponding portion of the image plane.

If the transmission unit is 1500 bytes, then, this example means thatthird packet, 10-th packet and 11-th packet, and 15-th packet are lost.The lacked information producing portion 258 derives a set of (headaddress, data length) from ID of such a lost packet, thereby producingthe lacked information 300. Then, by referring to the lacked informationconcerned, the decoding portion 260 substitutes the invalid data as thepixel value for the lacked portion, or the image analyzing portion 262differentiates the lacked information from other pieces of information,thereby carrying out the analysis.

FIG. 12 depicts another example of the data structure of the lackedinformation which is produced by the lacked information producingportion 258 of the information processing apparatus 200. This exampleexhibits the falling rate of the data every block obtained by dividingthe image plane. Specifically, the lacked information 304 is composed ofheader information 306 a, and a lacked information main body 306 b. Theheader information 306 a is composed of “the number of rows” and “thenumber of columns” of the blocks obtained by equally dividing an imageplane 308, and “the total data length” of the image.

In the depicted example, the image plane 308 is divided into 6 rows×8columns. In addition, the total data length is 1 M bytes. The number ofdivisions of the image plane is determined based on the analysiscontents, the properties of the photographed image, and the like. Thelacked information main body 306 b is composed of the coordinates of theblock including the “row” number, and the “column” number, “the datalength” of each of the blocks, and the “falling rate” of the data. Therow number and the column number, as depicted in the blocks of the imageplane 308, represent one block in pair of those. In the case where theimage plane is divided into 6 rows×8 columns, the row number is in therange of 0 to 5, and the column number is in the range of 0 to 7.

The data length of the block represents a size of the data contained ineach of the blocks and obtained by the compression coding. In the casewhere the variable length coding is carried out, as depicted in thefigure, the data length differs every block. The falling rate representsthe rate of the data lacked within the block. For example, the block of0 row×0 column represents that the data of 10% of the data of 20000bytes, that is, the data of 2000 bytes is lacked.

Since the lacked information depicted in FIG. 11 directly represents theaddress of the portion in which the data is lacked, the differentiationof the analysis processing can be carried out with the finer particlesize while as the number of intermittent lacks is further increased, thenumber of entries is increased, and the data size of the lackedinformation becomes large. With the lacked information 304 of FIG. 12,the degree of lack is grasped in area unit of the predetermined area,thereby enabling the data size of the lacked information to be madeconstant. For this reason, in the case where the information with thefine particle size is not regarded as important such as the case wherethe low resolution image is used in the image analysis, the format inthe figure becomes advantageous.

FIG. 13 depicts still another example of the data structure of thelacked information which is processed by the lacked informationproducing portion 258 of the information processing apparatus 200.Although similarly to the case of FIG. 12, this example represents thefalling rate of the data every block obtained by dividing the imagedata, as the division criteria of the block, instead of the area on theimage, the size of the coded data is fixed. That is, as depicted in astorage area 314 of the image data, the addresses are equally divided onthe storage area. In this case as well, the lacked information 310 iscomposed of header information 312 a, and a lacked information main body312 b.

The header information 312 a is composed of the “data length” of oneblock as the division unit, the “number of blocks” obtained through suchdivision, and the “total data length” of the image. The example depictedin the figure represents that the image data of 240 k bytes is dividedevery 10 k bytes, and as a result, the storage area 314 is divided into24 blocks. The data length of the division unit is determined based onthe analysis contents, the properties of the photographed image, and thelike.

The lacked information main body 312 b is composed of “head address” ofeach of the blocks, and “falling rate” of the data in each of theblocks. For example, the block in which the head address is “0”represents the data of 20% of the data of 10 k bytes, that is, the dataof 2000 bytes is lacked. Since similarly to the case of FIG. 11, thestorage area 314 of the image data is associated with the image plane,the data falling rate of each of the blocks in the storage area exhibitsthe data falling rate of the corresponding area of the image plane.

Since similarly to the case of FIG. 12, even in this example, the degreeof the lack is represented with the particle size larger than theparticle size in the transmission unit which is transmitted in onepacket, the size of the lacked information can be fixed irrespective ofthe lack situation. In addition, in the case where the variable lengthcoding is carried out, the division unit is prescribed by the data size,realizing in that in the area in which the data size after thecompression is larger, the area on the image corresponding to one blockbecomes small. Therefore, in the portion in which the image structure ismore complicated because the high-frequency components are contained,and so forth, the situation of the data lack can be reflected on onlythe analysis processing in the finer units.

Since the pieces of lacked information depicted in FIG. 11 to FIG. 13,as described above, have the respective properties, the optical form isselected in response to the structure of the space becoming thephotographing object, the contents of the image analysis to be carriedout, the resolution of the image used in the image analysis, theaccuracy required for the analysis, the characteristics of the hardwaresuch as the memory capacity and the process performance, and the like.For example, in the case where the detailed features need to beextracted by the face recognition or the like, it is considered toproduce the lacked information in the form depicted in FIG. 12 or FIG.13 in the case where a surface of a table or the like lying in the spaceto be photographed is roughly modulated in the form depicted in FIG. 11.

In the embodiment which has been described so far, the lackedinformation was produced for the portion for which the retransmissionrequest was made for the packet lost in the allowable time previouslydetermined, and the data could not be obtained even in such a manner. Bycarrying out the image analysis based on such a situation, even when theallowable time is restrictive, the analysis result as the best effect isacquired, and the analysis result which is considered to be low inaccuracy is treated so as to be differentiated from other causes,thereby enabling the influence to be prevented from being exerted on thewhole analysis result. Here, for the allowable time, such a time as thata shift between a motion in the photographed space, and a motion of thedisplay image is not recognized becomes the upper limit.

On the other hand, as depicted in FIG. 7, since as the allowable time isshorter, the falling rate of the data is increased, the shortening ofthe allowable time causes deterioration of analysis accuracy. FIG. 14 isa graphical representation explaining a relationship between theallowable time and the analysis accuracy. FIG. 14(a) exemplifies achange in analysis accuracy with respect to a change in data fallingrate, and FIG. 14(b), similarly to FIG. 7, exemplifies the data fallingrate with respect to a change in time which is allowed for theretransmission request. In the example depicted in FIG. 14(a), even whenthe data falling rate is increased from D1 to D2, a change in analysisaccuracy is relatively so small as to be depicted from A1 to A2. Thus,in the case where even when the analysis accuracy is at A2, a largeinfluence is not exerted on the information processing or display in thesubsequent stage, even the falling rate of D2 is allowed.

In this case, even when as depicted in FIG. 14(b), the allowable timefor the retransmission request is changed from L1 to L2, it may be saidthat a bad influence is not exerted on the information processing. Ifthe allowable time can be shortened in such a way, then, the currency ofthe display can be increased, and the more resources can be used inother information processing or transmission. Then, a mechanism foradjusting the allowable time for the retransmission request based on thechange in accuracy of the image analysis may be provided by utilizingsuch a relationship. Contrary to this, an algorithm for the imageanalysis may be optimized by using the given allowable time. Moreover,the parameter of the coding processing in the head mounted display 100may be adjusted in response to the optimized allowable time.

FIG. 15 depicts a configuration of functional blocks of the informationprocessing apparatus having functions of adjusting the retransmissionallowable time for the lost packet depending on the accuracy of theimage analysis. It should be noted that the blocks having the samefunctions as those of the information processing apparatus 200 depictedin FIG. 9 are assigned the same reference symbols and a descriptionthereof is suitably omitted here. An information processing apparatus200 a, similarly to the case of the information processing apparatus 200in FIG. 9, includes the communication portion 250, the packet analyzingportion 252, the buffer 256, the lacked information producing portion258, the decoding portion 260, the information processing portion 264,and the output data producing portion 266. The information processingapparatus 200 a further includes a retransmission request producingportion 254 a, an image analyzing portion 262 a, and a processingcondition adjusting portion 268.

The basic functions of the retransmission request producing portion 254a and the image analyzing portion 262 a are similar to those of theretransmission request producing portion 254 and the image analyzingportion 262 of the information processing apparatus 200 depicted in FIG.9. However, the image analyzing portion 262 a feeds back the accuracy ofthe image analyzing processing executed by itself to the processingcondition adjusting portion 268. After the processing conditionadjusting portion 268 associates the accuracy of the processingconcerned, and the lacked information of the data with each other, theprocessing condition adjusting portion 268 adjusts the set value of theallowable time so as to allow the falling rate in the range in which thenecessary accuracy is maintained. The retransmission request producingportion 254 a accepts the adjustment of such an allowable time.

When in the example depicted in FIG. 14, the lower limit allowed for theanalysis accuracy is A2, the processing condition adjusting portion 268reduces the allowable time in a direction toward L2 within the range ofthe falling rate in which the accuracy does not fall below A2. Forexample, at the timing of reception of a new frame, the set value of theallowable time is reduced by a predetermined increment ΔL, and carriesout the retransmission request within the allowable time concerned. Thereduction of the allowable time results in that the falling rate of thedata is increased. However, when the image of the new frame acquired insuch a manner is analyzed, the allowable time is further reduced unlessthe accuracy falls below A2. This processing is repetitively executeduntil the accuracy reaches A2, thereby obtaining the shorter value ofthe allowable time for which the necessary accuracy can be maintained.

Incidentally, in addition to the technique with which the allowable timeat a time point at which the allowable time is gradually reduced in sucha manner and the accuracy reaches the lower limit is set as the setvalue, there may be adopted such a technique that the allowable time ischanged in the large range to some extent to acquire a relationshipbetween the allowable time and the analysis accuracy, and thus theallowable time corresponding to the lower limit of the analysis accuracyis estimated. In any case, such adjustment processing is executed notonly at the time of the development or the manufacture of theinformation processing apparatus or the head mounted display, but alsoat the time of the operation of the system by the user.

That is, the shortest allowable time for which the suitable accuracy canbe maintained depends on not only the hardware such as the sensitivityof the image pickup element of the camera, the data correctionprocessing, the coding processing, and the communication environment ofthe inside of the camera, and the processing performance of theinformation processing apparatus, but also the situations such as thecontents of the image analysis, the brightness of the circumferences,the photographing conditions, and the quantity and colors of goods inthe photographed space. The reason for this is because the graphsdepicted in FIG. 14 are changed based on these conditions. Therefore,the allowable time is optimized at the various timing, resulting in thatwhile the influence exerted on the processing accuracy is kept to aminimum, the transmission environment can be optimized.

For example, when the user plays the electronic game by using theinformation processing apparatus 200 a, if in the initial stage of theelectronic game, the allowable time is adjusted in the background, then,it is possible to realize the processing under the optimal conditionsconforming to the contents of the game, and the actual operationenvironment. Incidentally, it is only necessary that the “analysisaccuracy” is some sort of index representing the strictness of the imageanalysis, and thus as depicted in FIG. 14, the analysis accuracy may notbe such a numerical value as to be continuously obtained.

That is, if a boundary for determining right and wrong of the allowabletime from a viewpoint of the accuracy can be set, the boundary may notbe given in the form of a numerical value. For example, two events as towhether or not the real object such as the hand which is essentially tobe detected is detected may be made the “analysis accuracy.” In thiscase, the turn of these events is the boundary, and thus it is onlynecessary to obtain the shortest allowable time in a state in which thereal object concerned is detected. The allowable time which is actuallyset may also be a value obtained by adding the predetermined margin tothe time acquired in such a manner. If the image analysis is theproduction of the depth image, then, the two events as to whether or notthe corresponding point is detected may be available, or a numericalvalue such as the image average of the degree of similarity when thecorresponding point is detected may also be available.

In addition, such analysis accuracy may also be acquired in the form ofa distribution in the image plane. As a result, the allowable time canbe more finely adjusted in response to the information as to where theanalysis accuracy becomes worse. For example, even when the accuracyfalls below the lower limit in the image edge having the low importance,if the accuracy is maintained at the more important image center in thearea in which the tracking object is present, and the like, then, itbecomes possible to further shorten the allowable time. If the allowabletime is optimized in such a way, then, the conditions for the variouskinds of pieces of processing in the head mounted display 100 may beadjusted in response to the optimization.

FIG. 16 depicts a configuration of functional blocks of a head mounteddisplay 100 having functions of adjusting the internal processingconditions based on the retransmission allowable time of the lostpacket. Incidentally, blocks having the same functions as those of thehead mounted display 100 depicted in FIG. 8 are assigned the samereference symbols, and a description thereof is suitably omitted here.The head mounted display 100 a, similarly to the case of the headmounted display 100 of FIG. 8, includes the data acquiring portion 120,the buffer 124, the retransmission control portion 128, and the dataoutputting portion 130. The head mounted display 100 a further includesa coding portion 122 a, a communication portion 126 a, and a processingcondition adjusting portion 132.

The basic functions of the coding portion 122 a and the communicationportion 126 a are similar to those of the coding portion 122 and thecommunication portion 126 depicted in FIG. 8. In this example, however,the parameter used in the compression coding is adjusted based on theallowable time for the packet retransmission. For this reason, theretransmission request producing portion 254 a of the informationprocessing apparatus 200 transmits the information associated with theallowable time which is finally decided by the processing conditionadjusting portion 268 to the head mounted display 100 a, and thecommunication Portion 126 a of the head mounted display 100 a receivesthat information. Then, the processing condition adjusting portion 132determines a coding parameter in response to the allowable timeconcerned, and sets the coding parameter in the coding portion 122 a.

The processing condition adjusting portion 132, for example, adjusts acompression ratio. Qualitatively, in such an unfavorable situation thatthe analysis accuracy cannot be obtained unless the allowable time islengthened to reduce the data falling rate, the compression ratio isincreased. As a result, if the size is reduced, then, the number ofpackets which shall be transmitted per one frame is reduced, and evenwith the same loss rate, the number of packets to be retransmitted canbe reduced. If the adjustment of the allowable time is carried out againunder such conditions, then, it is also possible that the allowable timecan be shortened.

Contrary to this, in a satisfactory situation in which even when theallowable time shortened, the analysis accuracy is obtained, thecompression ratio is lowered, thereby suppressing the deterioration ofthe image quality as well. The coding system itself may be switched overto a system in which the compression ratio is small and thedeterioration of the image quality is less. As a result, there issuppressed a load applied to the processing for the coding in the headmounted display 100 a. In the case where a power source is not suppliedto the head mounted display 100 a in a wired manner, the lightening ofthe processing load leads to an improvement as well in a battery lifetime due to the reduction of the power consumption.

Therefore, such a situation that the circumference is bright and theaccuracy of the image analysis is easy to obtain is specified based onthe optimized allowable time, and the compression ratio is adjusted inresponse to thereto, resulting in that the enhancement of the imagequality, the saving of the battery consumption, and the like can also beattained. After the processing condition adjusting portion 132 holdsinformation in which the allowable time, and the coded parameter or thecoding system are associated with each other in an internal memory, andspecifies the processing condition set in response to the allowable timetransmitted from the information processing apparatus 200 a, theprocessing condition adjusting portion 132 informs the coding portion122 a of this effect.

The coding portion 122 a carries out the compression coding of thephotographed image under the informed condition. It should be noted thatthe adjustment object by the processing condition adjusting portion 132is by no means limited to the coded parameter or the coding system. Forexample, in the case where the data acquiring portion 120 includes sucha function as to reduce the original image of the photographed imagestep by step and produce the photographed images having the differentresolutions in an Image Signal at Processor (ISP) or the like, theprocessing condition adjusting portion 132 may switch the resolution ofthe image produced in the ISP, or the resolution of the image whichshould be transmitted to the information processing apparatus 200 overto another one in response to the adjusted allowable time.

In this case, in such a situation that the analysis accuracy is notobtained unless as the allowable time after the adjustment is longer,that is, the allowable time is lengthened and the data falling rate ismade smaller, the image data having the low resolution is transmitted,thereby resulting in that the number of packets per one frame can bereduced to suppress the falling rate of the data. Then, if theadjustment of the allowable time is carried out again, then, it is alsopossible that the allowable time can be reduced.

Incidentally, as described above, by utilizing the fact that as thecircumference is brighter, the accuracy of the image analysis is easy toobtain, the measured value of the illuminance may be utilized in theadjustment of the allowable time in the information processing apparatus200 a. In this case, an luminance sensor (not depicted) is provided inthe head mounted display 100, and a measured value by the illuminancesensor is transmitted to the information processing apparatus 200. Theinformation processing apparatus 200 carries out the adjustment in sucha way that as the resulting illuminance is higher, the allowable time isshortened. From a diversified viewpoint, the optimal allowable time maybe obtained in combination with the adjustment based on the analysisaccuracy as described above.

Heretofore, the allowable time for the retransmission processing whichhas been described so far is set in the communication protocol, and isindependent of the application. This setting is enabled to be adjustedbased on the accuracy of the image analysis introduced by theapplication, thereby enabling a transmission aspect capable of flexiblycoping with the update of the hardware or the change in photographedenvironment to be presented. On the other hand, as approach in a reversedirection, the algorithm of the image analysis may be switched over toanother one in response to the allowable time set in accordance with thecommunication protocol.

In a word, the algorithm of the image analysis is switched over to suchan analysis algorithm that when the allowable time is given in FIG.14(b), the graph of FIG. 14(a) indicating the sufficient analysisaccuracy with the falling rate of the data corresponding to theallowable time is obtained. The algorithm with which even when thefalling rate is low, the analysis accuracy is obtained is qualitativelyconsidered to be an algorithm in which the sensitivity of the processingis low. For example, a filter for edge detection is switched from aSobel filter over to a Prewitt filter, next, a Roberts filter, resultingin that even when the influence of the data lacked portion can besuppressed, and the allowable time is insufficient, the accuracy can bemaintained to some extent. Contrary to this, since in the case where theallowable time is relatively long, the falling rate of the data isreduced to a low level, the algorithm having the high sensitivity isused, thereby enabling the more accurate result to be obtained.

In this case, after the processing condition adjusting portion 268 ofthe information processing apparatus 200 a holds the information inwhich the allowable time is associated with the analysis algorithm inthe internal memory, and specifies the algorithm corresponding to theallowable time set in the retransmission request producing portion 254a, the processing condition adjusting portion 268 informs the imageanalyzing portion 262 a of the effect. The image analyzing portion 262 aanalyzes the photographed image by using the informed algorithm. As aresult, even if the specification of the communication protocol orhardware is changed or upgraded, an influence of this situation can beprevented from being executed on the image analysis, and thus theinformation processing or the displayed contents.

Incidentally, it is also considered that the analysis algorithm is notonly switched so as to correspond to the allowable time given as thefixed value, but also associated with the allowable time adjusted inresponse to the environment or the like. For example, in the case wherethe mechanism for measuring the illuminance is provided in the manner asdescribed above, the allowable time is adjusted in response to theilluminance, and thus the algorithm suitable for the adjusted allowabletime can be selected.

Next, a description will now be given with respect to an operation ofthe information processing apparatus which can be realized by theconfiguration described above. FIG. 17 is a flow chart depicting aprocessing procedure for producing output data based on the photographedimage by the information processing apparatus in the embodiment. Itshould be noted that although the figure depicts the operation of theinformation processing apparatus 200 a including the function ofadjusting the allowable time described in FIG. 15, in the case wherethere is supposed the information processing apparatus 200 not includingthe adjustment function depicted in FIG. 9, the pieces of processing inS22 and S24 are omitted.

Firstly, when the user turns ON the power source of the informationprocessing apparatus 200 a and the head mounted display 100 to requestthe processing to start, the photographing is started by the camera 110of the head mounted display 100. The communication portion 250 of theinformation processing apparatus 200 a acquires the packet containingthe data associated with the first frame of the photographed image fromthe head mounted display 100 (S10). Since the data is transmitted in thetransmission unit step by step as described above, the processing in S10is actually executed in parallel with the subsequent processing. Thisalso applies to a retransmission request in S16 and lacked informationproducing processing in S18.

The packet analyzing portion 252 stores the data associated with thephotographed image in the buffer 256 in order of the acquisition of thepackets, and detects the lost packet by comparing the packet ID and thetransmission order with each other (S12). The retransmission requestproducing portion 254 a, at a timing at which the packet loss isdetected, starts to measure the elapsed time for the retransmission ofthe packet concerned. If the elapsed time is within the allowable timefor the retransmission (Y in S14), then, the retransmission requestproducing portion 254 a produces a retransmission request for the packetconcerned, and transmits the retransmission request to the head mounteddisplay 100 through the communication portion 250 (S16). With respect tothe lost packet, the pieces of processing in S10, S12, and S16 arerepetitively executed until the allowable time expires.

If the elapsed time exceeds the allowable time, then, the retransmissionrequest is not made (N in S14), then, the lacked information producingportion 258 produces the lacked information exhibiting that the datacorresponding to the packet concerned is lacked (S18). The decodingportion 260 and the image analyzing portion 262 a execute the decodingprocessing and the analysis processing, respectively, with the portionin the image in which the data is lacked being distinguished from otherportions while the lacked information concerned is referred (S20).Specifically, the decoding portion 260 acquires the data associated withthe photographed image from the buffer 256 and decodes the data thusacquired, and substitutes invalid data for the pixel value of theportion in which the data is lacked.

The image analyzing portion 262 a excludes the portion in which the datais lacked from the analysis object or the output object of the result,and produces additional information exhibiting that the portionconcerned has a possibility of the low accuracy of the analysis result.It should be noted that only one of the decoding portion 260 and theimage analyzing portion 262 a may cope with the data lack as describedabove depending on the format of the lacked information or the contentsof the image analysis.

In the case where the result of the image analysis proves that theaccuracy exceeds the predetermined lower limit (N in S22), theprocessing condition adjusting portion 268 adjusts the setting in theretransmission request producing portion 254 a so as to shorten theallowable time for the retransmission processing (S24). In the casewhere the accuracy reaches the predetermined lower limit (Y in S22), theadjustment for the allowable time is not carried out. It should be notedthat in the case where the accuracy falls below the lower limit from thebeginning, the adjustment may be carried out so as to lengthen theallowable time, thereby optimizing the analysis accuracy. In this case,however, from a viewpoint from the latency from the photographing to thedisplay, the upper limit for the allowable time needs to be especiallydecided.

The information processing portion 264 executes the informationprocessing by using the result of the image analysis or the like, andthe output data producing portion 266 produces the result of theinformation processing in the form of the output data associated withthe display image, the sound or the like (S26). The output dataconcerned is transmitted from the communication portion 250 to the headmounted display 100, and is outputted from the head mounted display 100.If there is no need for stopping the processing by the input or the likefrom the user, then, with respect to the subsequent frame of thephotographed image, the pieces of processing from S10 to S26 arerepetitively executed (N in S28). If there is caused the need forstopping the processing, then, all the pieces of processing are ended (Yin S28).

According to the embodiment which has been described so far, in thesystem for acquiring the data associated with the photographed imagethrough the wireless communication, and executing the informationprocessing, the lacked information is produced in which the data unableto be acquired is associated with the position on the image. Then, inthe image analysis or the information processing, the portion in whichthe data is lacked is differentiated from other portions to beprocessed. As a result, in the various pieces of processing until theoutput data is produced, the processing result of the portion in whichthe data is lacked is treated similarly to the case of the result ofother areas, and then the bad influence can be prevented from beingexerted on the entire output result.

In addition, such a measure as to execute the processing so as to avoidthe portion in which the data is lacked becomes possible depending onthe analysis contents. Therefore, the analysis accuracy for the area inwhich the data is normally obtained can be maintained, and theunnecessary processing is omitted, thereby enabling the load applied tothe processing to be lightened. These measures are taken for the datalack, thereby resulting in that even when a time allowable for theretransmission request responding to the packet loss during the datatransmission is limited, the accuracy for the processing can bemaintained as much as possible. As a result, the balance between theaccuracy of the processing result, and the currency can be madesatisfactory.

In addition, the allowable time, the analysis algorithm, the parameterof the coding processing, and the like are adjusted based on the uniqueknowledge about the relationship between the accuracy of the imageanalysis, and the time allowable for the retransmission request for thepacket. For example, the adjustment is carried out so as to shorten theallowable time within the range in which it is guaranteed to obtain theaccuracy, thereby realizing in that the influence exerted on the resultcan be reduced, and the speed up of the processing can be realized. Inaddition, the allowable time adjusted in such a manner reflects thecharacteristics of the photographed image that the analysis accuracy ishardly obtained. Therefore, the coding processing before thetransmission is optimized based on this situation, thereby enabling theoptimal transmission form responding to the photographing environment orthe like to be realized. For the switching of the analysis algorithmresponding to the allowable time, similarly, the processing under thecondition in which the best result is obtained for the given environmentincluding the communication environment can be executed.

The present invention has been described based on the embodiment so far.It is understood by a person skilled in the art that the embodiment ismerely an exemplification, and various modified changes can be made incombinations of these constituent elements or the processing processes,and such modified changes also fall within the scope of the presentinvention.

REFERENCE SIGNS LIST

30 . . . Display, 40 . . . Communication control portion, 44 . . .Antenna, 100 . . . Head mounted display, 110 . . . Camera, 120 . . .Data acquiring portion, 122 . . . Coding portion, 124 . . . Buffer, 126. . . Communication portion, 128 . . . Retransmission control portion,130 . . . Data outputting portion, 132 . . . Processing conditionadjusting portion, 200 . . . Information processing apparatus, 222 . . .CPU, 224 . . . GPU, 226 . . . Main memory, 250 . . . Communicationportion, 252 . . . Packet analyzing portion, 254 . . . Retransmissionrequest producing portion, 256 . . . Buffer, 258 . . . Lackedinformation producing portion, 260 . . . Decoding portion, 262 . . .Image analyzing portion, 264 . . . Information processing portion, 266 .. . Output data producing portion, 268 . . . Processing conditionadjusting portion.

INDUSTRIAL APPLICABILITY

As set forth hereinabove, the present invention can be utilized in thevarious kinds of information processing apparatuses such as the gamemachine, the image processing apparatus, and the personal computer, theinformation processing system including those, and the like.

The invention claimed is:
 1. An information processing apparatus,comprising: a communication portion configured to establish acommunication with an image pickup device and acquire data associatedwith a photographed image; a lacked information producing portionconfigured to produce lacked information in which data lacked as aresult of the communication is associated with a position on thephotographed image; an image analyzing portion configured todifferentiate a portion in which the data is lacked from other portionsby referring to the lacked information, thereby carrying out an analysisof the photographed image; and an output data producing portionconfigured to produce output data based on a result of the analysis andoutput the output data, wherein the lacked information producing portionincludes information exhibiting a ratio at which the data is lacked inevery block obtained by dividing an image into a predetermined area orby dividing image data into a predetermined data size.
 2. Theinformation processing apparatus according to claim 1, furthercomprising: a decoding portion configured to decode coded dataassociated with the photographed image acquired by the communicationportion, and substitute invalid data as a pixel value of the portion inwhich the data is lacked by referring to the lacked information.
 3. Theinformation processing apparatus according to claim 1, wherein thelacked information producing portion includes an address and a datalength of an area in which the data is lacked of a storage area of imagedata in the lacked information.
 4. The information processing apparatusaccording to claim 1, wherein the image analyzing portion excludes theportion in which the data is lacked, or an area in which a falling rateof the data is higher than a predetermined threshold value from ananalysis object based on the lacked information.
 5. The informationprocessing apparatus according to claim 1, wherein the image analyzingportion produces additional information associated with an effect thatwith respect to the portion in which the data is lacked, or an area inwhich a falling rate of the data is higher than a predeterminedthreshold value, analysis accuracy has a possibility of being low basedon the lacked information.
 6. An information processing systemcomprising: a head mounted display provided with an image pickup device;and an information processing apparatus configured to establish acommunication with the head mounted display, produce a display image andcause the head mounted display to display the display image; theinformation processing apparatus including a communication portionconfigured to acquire data associated with a photographed image from thehead mounted display, a lacked information producing portion configuredto produce lacked information in which data lacked as a result of thecommunication is associated with a position on the photographed image,an image analyzing portion configured to differentiate a portion inwhich the data is lacked from other portions by referring to the lackedinformation, thereby carrying out an analysis of the photographed image,and an output data producing portion configured to produce output databased on a result of the analysis and output the output data, whereinthe lacked information producing portion includes information exhibitinga ratio at which the data is lacked in every block obtained by dividingan image into a predetermined area or by dividing image data into apredetermined data size.
 7. An information processing method by aninformation processing apparatus, comprising: establishing acommunication with an image pickup device and acquiring data associatedwith a photographed image; producing lacked information in which datalacked as a result of the communication is associated with a position onthe photographed image; differentiating a portion in which the data islacked from other portions by referring to the lacked information,thereby carrying out an analysis of the photographed image; andproducing output data based on a result of the analysis and outputtingthe output data to a display device, wherein the lacked informationincludes information exhibiting a ratio at which the data is lacked inevery block obtained by dividing an image into a predetermined area orby dividing image data into a predetermined data size.